1. Document Purpose

This document is Part II of the benchmark specification. It describes the database structure and data set, test queries and test operational methodology.

It also contains a complete benchmark implementation, including scripts for database creation, tuning and data load, plus the OLTP and DSS queries that form the benchmark.

2. Database Structure and Data Set Characteristics

2.1. Base Table Schema

Our schema is composed of twelve data tables, each which use the following base structure:

Column Name	Data Type (Size)	Nulls?
Key	Unsigned Integer (4 bytes)	not null
Int	Unsigned Integer (4 bytes)	not null
Signed	Signed Integer (4 bytes)	nullable
Float	Single Precision Floating Point (4 bytes)	not null
Double	Double Precision Floating Point (8 bytes)	not null
Decim	Exact Decimal (18 bytes)	not null
Date	Date (8 bytes)	not null
Code	Fixed Length String (10 bytes)	not null
Name	Fixed Length String (20 bytes)	not null
Address	Variable Length String (2 to 80 bytes, 20 avg.)	not null

Table 1: Base Table Schema

The overall average row size is exactly 100 bytes.

If the tested database does not support types with the byte sizes indicated above, vendors may use alternative native types that provide no loss in precision. Any data type substitutions must have sizes as close to the sizes above, and, if necessary, pad remaining bytes with an unused data column to maintain an average row size of at least 100 bytes.

2.2. Base Table Data Distribution

There are four base table types (named uniques, hundred, tenpct and updates). Each use the above table structure but have different indices, data ranges and distributions of data.

Here are the data ranges, distribution and cardinality for each base table type. The high-level base table characteristics are:

Uniques: all columns have unique values

Hundred: most columns have exactly 100 unique values

Tenpct: most columns have 10% unique values

Updates: no special characteristics (used to test updates)

2.2.1. Uniques table type

Column Name	Value Range	Distribution	Cardinality
Key	0 to 1000000000, missing “1”	Sparse uniform	1
Int	0 to 1000000000, missing “1”	Sparse uniform	1
Signed	-500000000 to 500000000	Sparse uniform	1
Float	-500000000 to 500000000	Zipfian	1
Double	-1000000000 to 1000000000	Normal	1
Decim	-1000000000 to 1000000000	Sparse uniform	1
Date	1/1/1900 to 12/1/2000	Sparse uniform	1
Code	Alphanumeric	Sparse uniform	1
Name	Alphanumeric	Sparse uniform	1
Address	Alphanumeric	Sparse uniform	1

Table 2: Base Table Data Distribution to Uniques table type

2.2.2. Hundred table type

Column Name	Value Range	Distribution	Cardinality
Key	0 to no. of rows, missing “1”	Dense uniform	1
Int	0 to 1000000000 missing “1”	Sparse uniform	1
Signed	100 to 199	Dense uniform	100 unique values
Float	-500000000 to 500000000	Sparse uniform	100 unique values
Double	-1000000000 to 1000000000	Sparse uniform	100 unique values
Decim	-1000000000 to 1000000000	Sparse uniform	100 unique values
Date	1/1/1900 to 12/1/2000	Sparse uniform	1
Code	Alphanumeric	Sparse uniform	1
Name	Alphanumeric	Sparse uniform	100 unique values
Address	Alphanumeric	Sparse uniform	100 unique value

Table 3: Base Table Data Distribution to Hundred table type

2.2.3. Tenpct table type

Column Name	Value Range	Distribution	Cardinality
Key	0 to 1000000000, missing “1”	Sparse uniform	1
Int	0 to 1000000000, missing “1”	Sparse uniform	1
Signed	-500000000 to 500000000	Sparse uniform	1
Float	-500000000 to 500000000	Sparse uniform	10% unique values
Double	-1000000000 to 1000000000	Sparse uniform	10% unique values
Decim	-1000000000 to 1000000000	Sparse uniform	10% unique values
Date	1/1/1900 to 12/1/2000	Sparse uniform	1
Code	Alphanumeric	Sparse uniform	1
Name	Alphanumeric	Sparse uniform	10 unique values [sic]
Address	Alphanumeric	Sparse uniform	10% unique values

Table 4: Base Table Data Distribution to Tenpct table type

2.2.4. Updates table type

Column Name	Value Range	Distribution	Cardinality
Key	0 to no. of rows, missing “1”	Dense uniform	1
Int	0 to no. of rows, missing “1”	Dense uniform	1
Signed	-500000000 to 500000000	Sparse uniform	1
Float	-500000000 to 500000000	Zipfian	10 unique values [sic]
Double	-1000000000 to 1000000000	Normal	1
Decim	-1000000000 to 1000000000	Sparse uniform	1
Date	1/1/1900 to 12/1/2000	Sparse uniform	1
Code	Alphanumeric	Sparse uniform	1
Name	Alphanumeric	Sparse uniform	1
Address	Alphanumeric	Sparse uniform	1

Table 5: Base Table Data Distribution to Updates table type

2.3. Data Table Names and Sizes

The 11 data tables have the following names, base structure and sizes:

Table Name	Base Structure	Number of Rows
Uniques	Uniques	1,000,000
Hundred	Hundred	1,000,000
Tenpct	Tenpct	1,000,000
Updates	Updates	1,000,000
Tenthou	Tenpct	10,000
Hunthou	Hundred	100,000
Twomill	Updates	2,000,000
Threemill	Tenpct	3,000,000
Fourmill	Tenpct	4,000,000
Fivemill	Hundred	5,000,000
FourRAM	Uniques	20,000,000
Total Number of Rows:		38,110,000

Table 6: Data Table Names and Sizes

Sample DDL (in the form of SQL scripts) is included in the sample implementation code we are providing.

2.4. Data Import File Format

The data will be supplied as twelve comma-delimited ASCII text files. A CR/LF pair terminates each line. Note floating-point fields do not necessarily have decimal places and text fields are not quoted. Here is a sample row (word-wrapped to fit in these margins):

657665767,657665767,-146864686,334834835.00,-455455455,
-455455455,5/10/1918,HHX7lgNfzQ,P7saa6EArLThIOt:1lLi,
Hq82a5c1fO4Jn4dRuNCe6H1NfBQt8jaCZtuJlT

3. Required Table Constraints and Indices

Vendors must implement the constraints as shown below. Vendors must also create simple and compound indices on the fields indicated, and may not index any other fields or combination of fields than listed below (that is, the only allowable indexed fields are those listed below). This is restrictive, but to ensure each database has the exact same workload, we must ensure the same indices are present. A few fields are deliberately left unindexed to see how quickly the database under test can perform full table scans.

We do want to allow vendors that provide more index type choices to benefit from their engineering work. Pay careful attention to the characteristics of our data because vendors may choose the type of index (B*-tree, clustered, hashed, bitmap, etc.) used on any or all of the columns. Vendors may also add additional indices on the fields listed below (that is, a given field may be indexed twice using two different index types). Vendors may specify an index fill factor if desired.

Indexing decisions should be filed with implementation code.

Base Table Type	Constraints	Indices
Uniques	· Key is a Primary Key · Uniqueness constraint on Signed · Code is a Candidate Key	· Name
Hundred	· Key is a Primary Key · Code is a Candidate Key	· Int · Signed · Date · Name
Tenpct	· Key, Code is the compound Primary Key · Code is a Candidate Key	· Int · Signed · Float · Double · Decim · Date · Name
Updates	· Key is a Primary Key · Int is a Foreign Key (references Hundred.Key) · Code is a Candidate Key	· Signed · Decim · Date
Tenthou	· Key, Code is a compound Primary Key · Code is a Candidate Key	· Signed · Decim · Name
Hunthou		· Signed
Twomill	· Key is a Primary Key · Code is a Candidate Key	· Date
Threemill	· Key is a Primary Key	· Signed
Fourmill	· Key is a Primary Key	· Name
Fivemill	· Key is a Primary Key	· Code
FourRAM	· Key is a Primary Key
Upd_int_history	· Key, Int is a compound Primary Key
Oltp_write_01_check_value
Upd_signed_history	· Key, Signed is a compound Primary Key
Del_history	· Key is a Primary Key
Oltp_write_04_count_updates

Table 7:Required Table Constraints and Indices

3.1. Detailed Index Analysis

For a detailed index analysis showing table access patterns for each query, see the attached Index Analysis.xls spreadsheet.

4. Individual Test Details

4.1. OLTP Single Read Test

4.1.1. OLTP Single Read Test Description

OLTP Single Read Test measures how many single-row reads accessing a single table can be completed within the measurement window. This test measures absolute database performance on a single, simple query.

This is a multi-user test with a high number of users. The metrics are transactions per second (higher is better) and client response time (lower is better).

4.1.2. OLTP Single Read Test Workload

-- oltp_read_select_01 (randfourram)

-- single random row select on an index int field

-- uses: fourram

-- might use index on: fourram.p_key

-- returns 1 row

select p_key, p_code, p_date, p_signed, p_name

from fourram

where p_key = randfourram;

4.2. OLTP Read/Write Mix Test

4.2.1. OLTP Read/Write Mix Test Description

OLTP Read/Write Mix Test measures how many OLTP-type reads and writes accessing multiple tables can be completed within the measurement window. The insert, updates, deletes and selects range from very simple to moderately complex. This test measures database performance in a production OLTP environment.

This is a multi-user test with a high number of users. The metrics are transactions per second (higher is better) and client response time (lower is better).

4.2.2. OLTP Read/Write Mix Test Workload

-- oltp_read_select_01 (randfourram)

-- single random row select on an index int field

-- uses: fourram

-- might use index on: fourram.p_key

-- returns 1 row

select p_key, p_code, p_date, p_signed, p_name

from fourram

where p_key = randfourram

----------

-- oltp_read_select_02 (randbase)

-- single table random contiguous range select on an indexed int field, sort on an indexed character field

-- uses: hundred

-- might use index on: hundred.h_key, hundred.h_name (probably won't bother with the index with only 200 rows to sort)

-- returns 201 rows

select h_key, h_code, h_date, h_signed, h_name

from hundred

where h_key between randbase and randbase+200

order by h_name;

----------

-- oltp_read_select_03

-- single fixed row select on an indexed int field

-- uses: tenpct

-- might use index on: tenpct.t_key

-- returns 1 row

select t_key, t_int, t_signed, t_double, t_code, t_name

from tenpct

where t_key = 1001;

----------

-- oltp_read_select_04

-- single fixed row select on an indexed text field

-- uses: updates

-- might use index on: updates.p_code

-- returns 1 row

select p_key, p_int, p_signed, p_double, p_code, p_name

from updates

where p_code = 'BENCHMARKS';

----------

-- oltp_read_select_05

-- single table contiguous range select on an indexed int field

-- uses: updates

-- might use index on: updates.p_key

-- returns 100 rows

select p_key, p_int, p_signed, p_double, p_code, p_name

from updates

where p_key <= 100;

----------

-- oltp_read_select_06 (randbase)

-- single random row select on an indexed int field

-- uses: updates

-- might use index on: updates.p_int

-- returns 1 row

select p_key, p_int, p_signed, p_double, p_code, p_name

from updates

where p_int = randbase;

----------

-- oltp_read_select_07

-- single table contiguous range select on an indexed int field

-- uses: uniques

-- might use index on: uniques.u_key

-- returns 1 row

select u_key, u_int, u_signed, u_double, u_code, u_name

from uniques

where u_key <= 1000;

----------

-- oltp_read_select_08 (randbillion)

-- single table multiple index range select on an indexed character field and indexed int field

-- uses: tenpct

-- might use index on: tenpct.t_name, tenpct.t_key

-- returns ~1 rows (depends on random input)

select t_key, t_int, t_signed, t_double, t_code, t_name

from tenpct

where t_name = 'THE+ASAP+BENCHMARKS+'

and (t_key between randbillion and randbillion+10000);

----------

-- oltp_read_select_09

-- single single row select from a single table on an indexed float field

-- uses: tenpct

-- might use index on: tenpct.t_signed

-- returns 1 row (the minimum t_signed value in the table)

select t_key, t_int, t_signed, t_double, t_code, t_name

from tenpct

where t_signed <= -500000000;

----------

-- oltp_read_select_10

-- single table contiguous range select on an indexed float field, sort on an indexed int field

-- uses: tenthou

-- might use index on: tenthou.t_signed, tenthou.t_key

-- returns 2.5% of the rows (250 rows)

select t_key, t_int, t_signed, t_double, t_code, t_name

from tenthou

where t_signed <= -475000000

order by t_key;

----------

-- oltp_read_select_11

-- index scan (row doesn't exist in index)

-- uses: twomill

-- might use index on: twomill.p_key

-- returns 0 rows

select p_key, p_int, p_signed, p_double, p_code, p_name

from twomill

where p_key = 1;

----------

-- oltp_read_select_12

-- single table multiple index range select on an indexed character field and indexed float field, sort on an indexed int field

-- uses: tenpct

-- might use index on: tenpct.t_name, tenpct.t_signed

-- returns 0.01% of rows (108 rows)

select t_key, t_int, t_signed, t_double, t_code, t_name

from tenpct

where t_name like 'THE+%'

and t_signed > 499000000

order by t_key;

----------

-- oltp_read_select_13

-- single table range select and ordering on an indexed date field

-- uses: tenpct

-- might use index on: tenpct.t_date

-- returns ~0.05% of rows (~500 rows, or actually 439 rows)

select t_key, t_int, t_signed, t_double, t_code, t_name

from tenpct

where t_date > '12/15/1999'

order by t_date;

----------

-- oltp_read_select_14

-- single table range select on an indexed date field, sort on an indexed int field

-- uses: tenpct

-- might use index on: tenpct.t_date

-- returns 328 rows

select t_key, t_int, t_signed, t_double, t_code, t_name

from tenpct

where t_date between '01/10/1942' and '01/21/1942'

order by t_key;

----------

-- oltp_read_select_15

-- tablescan on an unindexed character field (row doesn't exist) -- will cause full tablescan

-- uses: uniques

-- no index used

-- returns 0 rows

select u_key, u_int, u_signed, u_double, u_code, u_name

from uniques

where u_name = 'xxxxxxxxxx';

----------

-- oltp_read_select_16

-- multiple row select from a single table on an indexed int field, sort on unindexed character and indexed float fields

-- uses: hunthou

-- might use index on: hunthou.h_address (this field deliberately not indexed), hunthou.h_signed

-- returns 100 rows

select distinct h_signed, h_address

from hunthou

where h_signed = 100

order by h_address;

----------

-- oltp_read_select_17

-- multiple row select and sort from a single table on an indexed fixed precision numeric field

-- uses: tenthou

-- might use index on: tenthou.t_decim

-- returns 10% of rows (1000 rows)

select distinct t_decim

from tenthou

order by t_decim;

----------

-- oltp_read_select_18

-- single table range select on an indexed int field, sort on an indexed character field and index int field

-- uses: tenthou

-- might use index on: tenthou.t_signed, tenthou.t_name

-- returns 1% of rows (100 rows)

select t_key, t_int, t_signed, t_double, t_code, t_name

from tenthou

where t_signed <= -490000000

order by t_name, t_key;

----------

-- oltp_read_join_01 (randbase)

-- single random row select on an indexed int field, two table equality join on unique indexed int fields

-- uses: updates, hundred

-- might use index on: updates.p_key, hundred.h_key

-- returns 1 row

select updates.p_key, updates.p_code, hundred.h_date, hundred.h_signed, hundred.h_name, hundred.h_address

from updates, hundred

where updates.p_key = randbase

and updates.p_key = hundred.h_key;

----------

-- oltp_read_join_02 (randtwomill)

-- single random row select on an indexed int field, two table equality join on indexed character fields

-- uses: twomill, uniques

-- might use index on: twomill.p_key, twomill.p_code, uniques.u_code

-- returns ~0 rows

select twomill.p_key, twomill.p_code, uniques.u_date, uniques.u_signed, uniques.u_name

from twomill, uniques

where twomill.p_key = randtwomill

and twomill.p_code = uniques.u_code;

----------

-- oltp_read_join_03 (randfivemill1, randfivemill2, randfivemill3, randfivemill4, randfivemill5)

-- single table random in (group) select, two table equality join on indexed int fields, order by on an indexed character field

-- uses: fivemill, updates

-- might use index on: fivemill.h_key, fivemill.h_code, updates.p_key

-- returns between 0-5 rows

select fivemill.h_key, fivemill.h_code, updates.p_date, updates.p_signed, updates.p_name

from fivemill, updates

where fivemill.h_key in (randfivemill1,randfivemill2,randfivemill3,randfivemill4,randfivemill5)

and fivemill.h_key = updates.p_key

order by fivemill.h_code;

----------

-- oltp_read_join_04 (randbase)

-- contiguous range select on an indexed int field, two table equality join on indexed int fields, dual sort on indexed character field, indexed date field

-- uses: updates, hundred

-- might use index on: updates.p_key, updates.p_code, hundred.h_key, hundred.h_date

-- returns 21 rows

select updates.p_key, updates.p_code,

hundred.h_date, hundred.h_signed, hundred.h_name, hundred.h_address

from updates, hundred

where (hundred.h_key between randbase and randbase+20)

and updates.p_key = hundred.h_key

order by updates.p_code, hundred.h_date;

----------

-- oltp_read_join_05

-- multiple table range select on indexed int fields, two table inequality join on indexed int fields

-- uses: uniques, tenpct

-- might use index on: uniques.u_key, tenpct.t_key

-- returns 11 rows

select uniques.u_key, uniques.u_name, tenpct.t_key, tenpct.t_signed

from uniques, tenpct

where uniques.u_key >= 1000000000

and tenpct.t_int <= uniques.u_int

and tenpct.t_key >= 999990000;

----------

-- oltp_read_join_06

-- multiple row select from a single table on an indexed character field; two table equality join on indexed int fields; sort on indexed float field (uniques.u_signed), unindexed character field (uniques.u_name), unindexed int field (fourram.p_signed), unindexed character field (fourram.p_name)

-- uses: uniques, fourram

-- might use index on: uniques.u_key, uniques.u_code, fourram.p_key

-- returns 2 rows

select distinct uniques.u_signed, uniques.u_name, fourram.p_signed, fourram.p_name

from uniques, fourram

where uniques.u_key = fourram.p_key

and uniques.u_code like 'AA%';

----------

-- oltp_read_join_07

-- single row select on an indexed int field, two table equality join on indexed int fields

-- uses: uniques, hundred

-- might use index on: uniques.u_key (x2), hundred.h_key

-- returns 1 row

select uniques.u_signed, uniques.u_name, hundred.h_signed, hundred.h_name

from uniques, hundred

where uniques.u_key = hundred.h_key

and uniques.u_key = 1001;

----------

-- oltp_read_join_08

-- single row select on an indexed character field, two table equality join on indexed character fields

-- uses: uniques, hundred

-- might use index on: uniques.u_code (x2), hundred.h_code

-- returns 1 row

select uniques.u_signed, uniques.u_name, hundred.h_signed, hundred.h_name

from uniques, hundred

where uniques.u_code = hundred.h_code

and uniques.u_code = 'BENCHMARKS';

----------

-- oltp_read_join_09

-- single row select on an indexed integer field, three table equality join on indexed integer fields

-- uses: uniques, hundred, tenpct

-- might use index on: uniques.u_key (x2), hundred.h_key, tenpct.t_key

-- returns 1 row

select uniques.u_signed, uniques.u_date, hundred.h_signed, hundred.h_date, tenpct.t_signed, tenpct.t_date

from uniques, hundred, tenpct

where uniques.u_key = hundred.h_key

and uniques.u_key = tenpct.t_key

and uniques.u_key = 1001;

----------

-- oltp_read_join_10

-- single row select on an indexed character field, three table equality join on indexed character fields

-- uses: uniques, hundred, tenpct

-- might use index on: uniques.u_code (x2), hundred.h_code, tenpct.t_code

-- returns 1 row

select uniques.u_signed, uniques.u_date, hundred.h_signed, hundred.h_date, tenpct.t_signed, tenpct.t_date

from uniques, hundred, tenpct

where uniques.u_code = hundred.h_code

and uniques.u_code = tenpct.t_code

and uniques.u_code = 'BENCHMARKS';

----------

-- oltp_read_join_11

-- single row select on an indexed int field, four table equality join on indexed int fields

-- uses: uniques, hundred, tenpct, updates

-- might use index on: uniques.u_key (x2), hundred.h_key, tenpct.t_key, updates.p_key

-- returns 1 row

select uniques.u_date, hundred.h_date, tenpct.t_date, updates.p_date

from uniques, hundred, tenpct, updates

where uniques.u_key = hundred.h_key

and uniques.u_key = tenpct.t_key

and uniques.u_key = updates.p_key

and uniques.u_key = 1001;

----------

-- oltp_read_join_12

-- single row select on an indexed character field, four table equality join on indexed character fields

-- uses: uniques, hundred, tenpct, updates

-- might use index on: uniques.u_code (x2), hundred.h_code, tenpct.t_code, updates.p_code

-- returns 1 row

select uniques.u_date, hundred.h_date, tenpct.t_date, updates.p_date

from uniques, hundred, tenpct, updates

where uniques.u_code = hundred.h_code

and uniques.u_code = tenpct.t_code

and uniques.u_code = updates.p_code

and uniques.u_code = 'BENCHMARKS';

----------

-- oltp_read_join_13

-- multiple contiguous range select from a single table on an indexed integer field, subselect on indexed float field using two table equality join on indexed integer fields

-- uses: uniques, tenpct

-- might use index on: uniques.u_key (x2), tenpct.t_signed, tenpct.t_key

-- returns 1027 rows

select uniques.u_int, uniques.u_name

from uniques

where uniques.u_key <= 2000000

and uniques.u_signed >

(select avg(tenpct.t_signed)

from tenpct

where uniques.u_key = tenpct.t_key);

----------

-- oltp_read_join_14

-- multiple contiguous range select from a single table on an indexed integer field, two table left outer equality join on indexed integer fields

-- uses: tenthou, tenpct

-- might use index on: tenthou.t_code, tenthou.t_key, tenpct.t_code

-- returns 10 rows

select tenthou.t_key, tenthou.t_name, tenthou.t_signed, tenthou.t_code, tenpct.t_key

from tenthou left outer join tenpct

on (tenthou.t_code = tenpct.t_code)

where tenthou.t_key <= 1000000;

----------

-- oltp_write_01 (randbase)

-- increment randomly chosen updates.p_int by 1 and save old row in update_int_history

-- changes: updates.p_int, update_int_history

-- might use index on: updates.p_key

-- competes with: oltp_read_select_04, oltp_read_select_05, oltp_read_select_06, oltp_write_06

-- checks atomicity, isolation

-- single table 1 row select and update on an int, 1 row insert

-- stored procedure code is database specific – see sample implementation code

----------

-- oltp_write_02 (randbase)

-- increment randomly chosen updates.p_signed by 1 and save old row in update_signed_history

-- changes: updates.p_signed, update_signed_history

-- might use index on: updates.p_key

-- competes with oltp_read_select_04, oltp_read_select_05, oltp_read_select_06, oltp_read_join_03

-- checks atomicity, isolation

-- single table 1 row select and update on an int, 1 row insert

-- stored procedure code is database specific – see sample implementation code

----------

-- oltp_write_03 (randfivemill)

-- moves a randomly selected row from fivemill to del_history

-- changes: fivemill, del_history

-- might use index on: fivemill.h_key

-- competes with oltp_read_join_03

-- checks atomicity, isolation

-- single row select and insert, single row delete

-- stored procedure code is database specific – see sample implementation code

----------

-- oltp_write_04 (randfourmill)

-- modifies between 1 and 2 randomly chosen fourmill.t_name and fourmill.t_address values, increments the update counter oltp_write_04_count_updates.num_updates

-- changes: fourmill.t_name, fourmill.t_address, oltp_write_04_count_updates.num_updates

-- might use index on: fourmill.t_key

-- competes with nothing else

-- checks atomicity

-- single table in range select and update on 2 character fields, 1 int field

-- stored procedure code is database specific – see sample implementation code

----------

-- oltp_write_05 (randbase)

-- does an update on 10 contiguous rows of updates.p_decim and then rolls the changes back

-- changes: (nothing)

-- might use index on updates.p_key

-- competes with oltp_write_06, dss_select_05, dss_select_06, dss_select_07, dss_select_08, dss_select_09

-- checks atomicity, consistency, isolation

-- single table range update and rollback

-- stored procedure code is database specific – see sample implementation code

4.3. Mixed Workload Test

4.3.1. Mixed Workload Test Description

Mixed Workload Test measures how many (read/write) OLTP- and (read-only) DSS-type transactions accessing multiple tables can be completed within the measurement window. The mix is heavily weighted towards OLTP transactions, but all transactions need to be treated equally by the database (no preferential treatment is allowed for OLTP transactions). This test measures database performance in a production OLTP environment when a small amount of concurrent ad hoc analysis queries are also included.

This is a multi-user test with a high number of users. The metrics are transactions per second (higher is better) and client response time (lower is better).

4.3.2. Mixed Workload Test Workload

This test includes all the queries in the OLTP Read/Write Mix Text plus the following queries:

-- dss_select_01

-- uses: tenpct

-- might use index on: tenpct.t_key

-- returns 1 row, answer is "999999"

select count(t_key)

from tenpct

where t_key <> 0;

----------

-- dss_select_02

-- uses: uniques

-- might use index on: uniques.u_key

-- returns 1 row, answer is "0"

select min(u_key)

from uniques;

----------

-- dss_select_03

-- uses: hundred

-- might use index on: hundred.h_key, hundred.h_name

-- returns 100 rows

select max(h_key)

from hundred

group by h_name

order by 1;

----------

-- dss_select_04

-- uses: tenpct

-- might use index on: tenpct.t_name, tenpct.t_int, tenpct.t_signed, tenpct.t_float, tenpct.t_double, tenpct.t_decim

-- returns 1 row, answer is "0"

select count(t_key)

from tenpct

where t_name = 'THE+ASAP+BENCHMARKS+'

and t_int <= 100000000

and t_signed between 1 and 99999999

and not (t_float between -450000000 and 450000000)

and t_double > 600000000

and t_decim < -600000000;

----------

-- dss_select_05

-- uses: updates, hundred

-- might use index on: updates.p_key, updates.p_decim, hundred.h_key

-- returns 1 row, answer is "995000995"

select avg(updates.p_decim)

from updates

where updates.p_key in

(select updates.p_key

from updates, hundred

where hundred.h_key = updates.p_key

and updates.p_decim > 990000000);

----------

-- dss_select_06

-- uses: reportview (updates, hundred)

-- might use index on: updates.p_key, reportview.r_decim (updates.p_decim), hundred.h_key, reportview.r_code (hundred.h_code), reportview.r_int (hundred.h_int) (last two are unlikely, a simple row scan is a better choice)

-- returns 10,000 rows

select avg(r_signed), min(r_signed), max(r_signed), min(r_date), max(r_date), count(distinct r_name), count(r_name)

from reportview

where r_decim > 980000000

group by r_code, r_int

order by 1;

----------

-- dss_select_06_noview (use this query if the database doesn’t support views)

-- uses: updates, hundred

-- might use index on: updates.p_key, updates.p_decim, hundred.h_key, hundred.h_code, hundred.h_int

-- returns 10,000 rows

select avg(updates.p_signed), min(updates.p_signed), max(updates.p_signed), min(hundred.h_date), max(hundred.h_date), count(distinct hundred.h_name), count(hundred.h_name)

from updates, hundred

where updates.p_key = hundred.h_key

and updates.p_decim > 980000000

group by hundred.h_code, hundred.h_int

order by 1;

----------

-- dss_select_07

-- uses: reportview (updates, hundred)

-- might use index on: updates.p_key, reportview.r_decim (updates.p_decim), hundred.h_key

-- returns 1 row

select min(r_signed), max(r_signed), min(r_date), max(r_date), count(distinct r_name), count(r_name), count(r_code), count(r_int)

from reportview

where r_decim > 900000000;

----------

-- dss_select_07_noview (use this query if the database doesn’t support views)

-- uses: updates, hundred

-- might use index on: updates.p_key, updates.p_decim, hundred.h_key

-- returns 1 row

select min(updates.p_signed), max(updates.p_signed), min(updates.p_date), max(updates.p_date), count(distinct hundred.h_name), count(hundred.h_name), count(hundred.h_code), count(hundred.h_int)

from updates, hundred

where updates.p_key = hundred.h_key

and updates.p_decim > 900000000;

----------

-- dss_select_08

-- uses: hundred

-- might use index on: hundred.h_name

-- returns 100 rows, all counts are "10000"

select h_name, count(h_name)

from hundred

group by h_name

order by 2;

----------

-- dss_select_09

-- uses: hundred

-- might use index on: hundred.h_name

-- returns 100 rows

select h_name

from hundred

group by h_name

having count(h_name) > 10

order by h_name;

----------

-- dss_select_10

-- uses: hunthou

-- might use index on: hunthou.h_float (will need to tablescan anyway)

-- returns 1 row, answer is "50000"

select count(h_name)

from hunthou

where h_float >

(select avg(h_float) from hunthou);

----------

-- dss_select_11

-- uses: threemill

-- might use index on: threemill.t_signed

-- select 25% of the table

-- uses: threemill

-- might use index on: threemill.t_signed

-- returns 1 row, answer is "750000"

select count(t_key)

from threemill

where t_signed <= -250000000;

4.4. Dedicated Reporting Test

4.4.1. Dedicated Reporting Test Description

Dedicated Reporting Test measures how fast the database can process a given set of DSS-type queries submitted in a fixed order when it is doing no other work. The queries range from simple to very demanding. This test measures database performance in a dedicated reporting environment.

This is a single-user test. The metric is total completion time and faster is better.

4.4.2. Dedicated Reporting Test Workload

The test uses just the DSS Select queries in the Mixed Workload Test.

4.5. Online Reporting Test

4.5.1. Online Reporting Test Description

Online Reporting Test measures throughput and response time of the OLTP Read/Write Mix Test at a fixed user load while the database is also running the Reporting Test above.

This is a multi-user test with a moderate number of users. The metrics are OLTP transactions per second (higher is better) and OLTP client response time (lower is better). We will also be measuring completion time for the report generation run, but this metric isn’t as important to us in this test as maintaining OLTP performance.

4.5.2. Online Reporting Test Workload

The test uses the queries from the OLTP Read/Write Mix Text and the Reporting Test.

4.6. Online Backup Test

4.6.1. Online Backup Test Description

Online Backup Test measures throughput and response time of the OLTP Read/Write Mix Test at a fixed user load while the database also performs a full database backup (data, indices and transaction log). We will be using a single DLT tape drive as a backup target.

This is a multi-user test with a moderate number of users. The metrics are OLTP transactions per second (higher is better) and OLTP client response time (lower is better). We will also be measuring completion time for the database backup job, but this metric isn’t as important to us in this test as maintaining OLTP performance.

4.6.2. Online Backup Test Workload

This test uses the queries from the OLTP Read/Write Mix Test.

4.7. Other Tests

4.7.1. Load and Index Test Description

Load and Index Test measures how long it takes for the database to load, index and update optimizer statistics for locally stored test data set stored as ASCII flat files. Vendors are can load, index and update statistics separately or can combine these operations if their database provides this option. The metric is total time and faster is better.

Because of variations in different database data loaders, vendors should load, index and update statistics for a single table (the fourtymill table) as separate operations, each timed separately. Once this is complete, the table will be dropped and the entire database will then be loaded, indexed and have statistics updated using the fastest mechanism the vendor provides. We will report this entire process as one figure since some products combine these steps into a single procedure.

Do not enforce constraints such as uniqueness or referential integrity on load. This will be done as part of the index creation step.

Take maximum advantage of the SMP capabilities of the server hardware during this test.

This test can be implemented either using a DLL function doing a remote procedure call to the server or as server-side scripts.

4.7.2. Load and Index Test Description

This code is database specific – see sample implementation code.

4.7.3. Update Statistics Test Description

Update Statistics Test measures how long it takes to update optimizer statistics for the updates and fourmill tables (using a full table scan—no sampling) after the OLTP Read/Write Mix Test has been run. The metric is total time and faster is better.

4.7.4. Update Statistics Test Workload

This code is database specific – see sample implementation code.

5. SQL Coverage Map

Here is a grouping of the various SQL queries into functional categories.

5.1. Select with Equality (integer)

oltp_read_select_01

oltp_read_select_03

oltp_read_select_06

oltp_read_select_16

oltp_read_join_01

oltp_read_join_02

oltp_read_join_07

oltp_read_join_09

oltp_read_join_11

oltp_write_01

oltp_write_02

oltp_write_03

oltp_write_06

5.2. “In” a set (integer)

oltp_read_join_03

5.3. Select with NOT Equality (integer)

dss_select_01

5.4. Select with Equality (char)

oltp_read_select_04

oltp_read_select_08

oltp_read_join_08

oltp_read_join_10

oltp_read_join_12

dss_select_04

5.5. Select with Between (integer)

oltp_read_select_02

oltp_read_select_08

oltp_read_join_04

oltp_write_04

oltp_write_05

5.6. Select with Between (float)

dss_select_04

5.7. Select with Between (date)

oltp_read_select_14

5.8. Select with Single Inequality (integer)

oltp_read_select_05

oltp_read_select_07

oltp_read_select_09

oltp_read_select_10

oltp_read_select_12

oltp_read_select_18

oltp_read_join_05

oltp_read_join_13

oltp_read_join_14

dss_select_04

dss_select_05

dss_select_06 / dss_select_06_noview

dss_select_07 / dss_select_07_noview

dss_select_09

dss_select_11

5.9. Select with Single Inequality (float)

dss_select_04

dss_select_10

5.10. Select with Single Inequality (date)

oltp_read_select_13

5.11. Select with Like (char)

oltp_read_select_12

oltp_read_join_06

5.12. Sort (integer)

oltp_read_select_10

oltp_read_select_12

oltp_read_select_14

oltp_read_select_18

dss_select_08

5.13. Sort (exact decimal)

oltp_read_select_17

5.14. Sort (float)

dss_select_06 / dss_select_06_noview

dss_select_07 / dss_select_07_noview

5.15. Sort (char)

oltp_read_select_02

oltp_read_select_16

oltp_read_select_18

oltp_read_join_03

oltp_read_join_04

dss_select_09

5.16. Sort (date)

oltp_read_select_13

oltp_read_join_04

5.17. Sort/Merge (integer) (Select Distinct or Group By)

oltp_read_select_17

dss_select_03

dss_select_06 / dss_select_06_noview

dss_select_07 / dss_select_07_noview

5.18. Sort/Merge (char) (Select Distinct or Group By)

dss_select_06 / dss_select_06_noview

dss_select_07 / dss_select_07_noview

dss_select_08

dss_select_09

5.19. Full index scan

oltp_read_select_11

5.20. Full table scan

oltp_read_select_15

5.21. Equality Join (two tables)

oltp_read_join_01

oltp_read_join_02

oltp_read_join_03

oltp_read_join_04

oltp_read_join_06

oltp_read_join_07

oltp_read_join_08

oltp_read_join_13

dss_select_05

dss_select_06 / dss_select_06_noview

dss_select_07 / dss_select_07_noview

5.22. Equality Join (three tables)

oltp_read_join_09

oltp_read_join_10

5.23. Equality Join (four tables)

oltp_read_join_11

oltp_read_join_12

5.24. Inequality Join (two tables)

oltp_read_join_05

5.25. Left Outer Join

oltp_read_join_14

5.26. Average

oltp_read_join_13

dss_select_05

dss_select_06 / dss_select_06_noview

dss_select_07 / dss_select_07_noview

dss_select_10

5.27. Count

dss_select_01

dss_select_04

dss_select_06 / dss_select_06_noview

dss_select_07 / dss_select_07_noview

dss_select_08

dss_select_09

dss_select_10

dss_select_11

5.28. Min/Max

dss_select_02

dss_select_03

dss_select_06 / dss_select_06_noview

dss_select_07 / dss_select_07_noview

6. Results Verification

To check that queries are returning the correct number of rows and the correct values in those rows, see the attached Query Analysis.xls spreadsheet.

There are also a series of data integrity checks included in the attached sample benchmark implementation code.

7. Workload Distributions

Exact query workload distributions for all of the tests above are in the attached Workload Mix Distributions.xls spreadsheet.

8. Test Run Times

8.1. OLTP Test Run Times

For OLTP tests, we will be using the following timings:

Start Time	End Time	Activity	Duration
(Not timed)	0:00	Clients log onto database.
0:00	6:00	Warm up (ramp up) time to allow the cache to fill, queries to be compiled, execution plans to be generated, etc.	6:00
6:01	10:00	Measurement (execution) time.	4:00
10:01	10:30	Warm down (ramp down) time; allow currently executing procedures to finish.	1:30
10:31	11:00	Quiet time; allow statistics gathering.	0:30
11:01	(Not timed)	Clients log off database.
Total Time:			~12:00

Table 8: Test Run Times for OLTP tests

8.2. Mixed Workload Run Times

Because of the longer-running queries it contains, the Mixed Workload Test uses a longer timing scheme to ensure enough queries are submitted to maintain our desired mix distribution.

Start Time	End Time	Activity	Duration
(Not timed)	0:00	Clients log onto database.
0:00	10:00	Warm up (ramp up) time to allow the cache to fill, queries to be compiled, execution plans to be generated, etc.	10:00
10:01	20:00	Measurement (execution) time.	10:00
20:01	21:00	Warm down (ramp down) time; allow currently executing procedures to finish.	1:30
21:01	21:30	Quiet time; allow statistics gathering.	0:30
21:31	(Not timed)	Clients log off database.
Total Time:			~22:30

Table 9: Test Run Times for Mixed Workload Test

Benchmark control logic ensures that all clients are logged into the database before the warm up time begins and that all transactions have completed and all clients are logged off before the next benchmark starts.

8.3. Think Times

Transactions will have no think time between them.

9. Benchmark Factory Settings

· In the Project, Settings, General page, “AutoStart Project Wizard on New” should be OFF, “Verify the number of Agents is the same as the testbed configuration” should be OFF and “Ignore Errors from Server” should be ON.

· Set the multiuser test timing durations to the figures in the “Test Run Times” section (above).

· Set the multiuser test user loads to the figures in the “Test User Loads” section of part I of this specification (for most tests, the user load figures are 1, 25, 50, 75, 100, 200 and 300 users). The Online Reporting and Online Backup tests run at 300 users.

· In each test’s Properties, Options page, ensure “Ignore Errors from Server” is ON. Set “Save Transaction Statistic Summary to Repository” to ON, uncheck the “Perform checkpoint at start of each test iteration” option and uncheck the “Perform checkpoints during the timed period of the test” option.

· Note that when using the custom .DLL files we provide, don’t use Benchmark Factory's New Project wizard to set up the database. The benchmark database needs to be built using the separate scripts provided, not through Benchmark Factory.

10. Post-Benchmark Activities

· Copy the SQL Anywhere test repository files (in C:\Program Files\CSS\Adaptive Server Anywhere\data) to a private storage location. If more testing is to occur, restore the database repository to a clean configuration by manually deleting all the data from the database or by copying empty database files into the SQL Anywhere data folder.

· Copy C:\Program Files\CSS\Benchmark Factory\Error Logs to storage location and delete all the files in that directory.

· Copy C:\Program Files\CSS\Benchmark Factory\Result Logs to storage location and delete all the files in that directory.